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Background / Context: 

Description of prior research and its intellectual context. 

Closing the academic achievement gap between advantaged and disadvantaged students is a critical focus 
for federal, state, and local policymakers. Despite improvements in academic achievement over the past 
few decades, proficiency gaps still persist across income, racial, and ethnic groups (National Center for 
Education Statistics, 2013). Expanded Learning Time (ELT) is seen as one promising approach for 
improving academic achievement, particularly in the nation’s chronically low-performing schools. 

In recent years, the number of schools across the country that are implementing ELT has increased. 
According to the National Center for Time and Learning (NCTL) database*, there are currently 1,033 
ELT schools located in 47 states and the District of Columbia. The states with the highest number of ELT 
schools are North Carolina (106), California (99), New Jersey (94), Massachusetts (91), and Illinois (87). 
While a majority of current ELT schools (60%) are charter schools, ELT programs are increasingly being 
considered and adopted by traditional public schools as well (National Center on Time and Learning 
(NCTL), 2012). 

Since 1995, Citizen Schools (CS) has developed and supported the implementation of its own ELT 
model. The CS ELT model use an additional shift of educators and community volunteers to engage 
middle school students in hands-on apprenticeships, while simultaneously providing individualized 
supports to ensure academic and future success. In the past five years, CS has tripled the number of 
students served and expanded its geographic presence. 

ELT implementation is hypothesized to lead to a number of desired outcomes for students. A recent 
review of the literature found that although designs are generally weak for making causal inferences, 
extending school time can be an effective way to support student achievement, particularly for 
disadvantaged students and when attention is paid to how time is used (Patall, Cooper, and Allen, 2010; 
Valentine, Cooper, Patall, Tyson, & Civey Robinson, 2010). Theory suggests that as a result of ELT, 
students may become more engaged in school because of additional enrichment and learning 
opportunities, develop better communication skills due to more time with teachers and peers, and be less 
likely to engage in disruptive behavior because of less idle time. Ultimately, student achievement may 
improve as a result of additional learning time. The authors are conducting an evaluation of the CS ELT 
model that includes an implementation and outcomes component. 


Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

Over the course of the ELT evaluation, it has become and remains evident that implementation varies 
substantially across ELT campuses. Identifying strengths and weaknesses in program implementation is a 
key component to understanding the sources of such variation and the relationship to student achievement 


* National Center on Time and Learning (NCTL): Expanded-Time Schools Database. Retrieved from 
http://www.timeandlearning.org/db/. 
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results. To this end, an implementation index was developed that examines seven core program elements 
of the ELT model. The index draws from interviews and surveys, and incorporates information from 
multiple school-based respondents, including principals, classroom teachers. Citizen Schools Campus 
Directors and Citizen Schools fellows. 

Setting: 

Description of the research location. 

Middle schools located in several districts and states across the country. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

Study participants include middle school students from schools in MA, NY, NJ, TX, IL, NM, CA serving 
predominantly low-income (90% nationally), racial or ethnic minority (94% nationally), and academically 
struggling students. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

The ELT initiative mobilizes a second shift of educators for two to three hours per day (four days/week), 
who provide academic support, leadership development, and “apprenticeships” after the end of the 
traditional school day. All students in grades designated for the ELT program (if not school-wide) are 
required to participate in ELT programming. 

Research Design: 

Description of the research design. 

The study is a five-year evaluation with both implementation and impact components to assess the 
operation and effects of ELT programming on middle school students. The impact study uses a quasi- 
experimental comparative interrupted time series (CITS) design that leverages pre-program data and data 
from matched comparison schools to produce estimated effects representing differences between ELT and 
comparison schools beyond what one might expect given pre-program measures and other secular 
initiatives affecting all schools. The implementation component of the study incorporates surveys, 
interviews, and site visits to assess how schools integrate ELT into their school days, organizationally and 
instructionally. This paper focuses specifically on how to characterize implementation systematically and 
in such a way as to explain variation in student achievement results. 

The multi-dimensional index used to detect program implementation variation includes seven key 
elements of the Citizen Schools program model: 

1) planning; 

2) leadership; 

3) data collection; 

4) training and professional development; 

5) family/community engagement; 

6) alignment/coordination between partner school and CS; and 

7) 4 pillars of the Citizen Schools ELT model. 
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Each program element is measured by a number of related indicators based on survey and interview data. 
For each program element, the levels of fidelity reflect increasing and corroborated evidence that the core 
element has been implemented as intended. The index also measures the congruence — or level of 
agreement — among diverse sets of respondents about each specific element. Scores are based upon the 
presence or absence of the structural elements of the implementation index, and do not represent a 
measure of the perceived or observed quality 


Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 


The data collection methods and schedule are illustrated below. 



Study Year (School Year) 


1 (2010-11) 

2 (2011-12) 

3 (2012-13) 

4 (2013-14) 

Site Visit (ELT only) 

V 


V 


Principal Telephone Interview (ELT) 


V 


✓ 

Principal Telephone Interview (MC) 


V 



Survey 

^Student (ELT & MC) 


V 

V 


**Teacher (ELT & MC) 


V 

vZ 

(ELT only) 

CS staff (ELT only) 


V 

V 

V 

Extant (Test Score) Data 


V 

V 



Findings / Results: 

Description of the main findings with specific details. 

Preliminary results about implementation indicate Cohort 1 and 2 schools recorded similar average total 
evidence scores in years 1 and 2 (16.8 and 16.6, respectively out of a possible 21); Cohort 1 schools 
scored slightly higher in year 3, on average (17). Additionally, most schools exhibited strong fidelity of 
implementation for the planning, leadership, and data collection elements, and lower fidelity of 
implementation for alignment/coordination between Citizen Schools and partner schools, training and 
professional development, and family and community engagement elements. 
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Planning Leadership Data Training Community Alignment 4 Pillars 

Collection 

Program Elements 


■ Level 1 ■ Level 2 Level 3 


The study team is currently completing analyses (October 2013) to assess relationships between 
implementation and outcomes. 


Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

While the analyses of student achievement have not yet been completed, analysis of implementation 
indicates substantial variation across schools. The authors plan to conduct relational analyses examining 
associations between levels of implementation and student achievement results, to learn whether (and if 
so, how) efforts to measure the implementation levels of measurable programmatic constructs can inform 
our understanding of specific mechanisms that may influence student achievement. 


References: 

National Center on Time and Learning (2012). Mapping the Field: A Report on Expanded-Time Schools 
in America. National Center on Time and Learning, Boston, MA. 

Patall, E., H. Cooper, and A. Allen. "Extending the School Day or School Year: A Systematic Review of 
Research (1985-2009)." Review of Educational Research (2010): 401-436. 

Valentine,}. C., Cooper, H., Patall, E. A., Tyson, D, & Civey Robinson,}. (2010). A Method for Evaluating 
Research Syntheses: The Quality, Conclusions, and Consensus of Twelve Syntheses of the Effects of 
After School Programs. Research Synthesis Methods, 1, 20-38. 


SREE Spring 2014 Conference Abstract Template 


B-4 



